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ABSTRACT 

In  December  2002,  the  US  Government  implemented  policy  to  immunize  health 
workers,  first  responders  and  military  personnel  against  smallpox  in  preparation  for 
a  possible  bioterrorist  attack.  Self-reported  vaccination  data  are  commonly  used  in  epide¬ 
miologic  research  and  may  be  used  to  determine  vaccination  status  in  a  public  health 
emergency.  To  establish  a  measure  of  reliability,  the  agreement  between  self-reported 
smallpox  vaccination  and  electronic  vaccination  records  was  examined  using  data  from 
the  Millennium  Cohort  Study.  Descriptive  measures  and  a  kappa  statistic  were  calculated 
for  data  from  54,066  Millennium  Cohort  Study  participants.  Multivariable  modeling 
adjusting  for  potential  confounders  was  used  to  investigate  vaccination  agreement  status 
and  health  metrics,  as  measured  by  the  Short  Form  36-Item  Health  Survey  for  Veterans 
(SF-36V)  and  hospitalization  data.  Substantial  agreement  (k  =  0.62)  was  found  between 
self-report  and  electronic  recording  of  smallpox  vaccination.  Of  all  participants  with 
an  electronic  record  of  smallpox  vaccination,  90%  self-reported  being  vaccinated; 
and  of  all  participants  with  no  electronic  record  of  vaccination,  82%  self-reported  not 
receiving  a  vaccination.  There  was  no  significant  difference  in  hospitalization  experience 
prior  to  questionnaire  completion  between  vaccinated  and  unvaccinated  participants. 
While  overall  scores  on  the  SF-36V  suggested  a  healthy  population,  participants 
whose  self-reported  vaccination  status  did  not  match  electronic  records  had  slightly 
lower  adjusted  mean  scores  for  some  scales.  These  results  indicate  strong  reliability 
in  self-reported  smallpox  vaccination  and  also  suggest  that  discordant  reporting 
of  smallpox  vaccination  is  not  associated  with  substantial  differences  in  health  among 
Millennium  Cohort  participants. 


INTRODUCTION 

The  World  Health  Organization  announced  in  1980  that  naturally  occurring  smallpox 
had  been  eradicated  worldwide  following  the  last  case  reported  in  Somalia  in  1977.1,2 
Once  the  risk  of  natural  infection  had  disappeared,  routine  vaccination  was  discontinued. 
Laboratories  with  existing  smallpox  (variola  virus)  were  requested  to  destroy  the  virus 
or  turn  their  stocks  over  to  one  of  two  collaborating  centers.2,3  It  has  been  suggested, 
however,  that  the  variola  virus  may  have  been  retained  for  use  as  a  biological  weapon.4 
In  the  wake  of  the  September  11,  2001  terrorist  attacks,  the  US  Government  began 
to  immunize  health  workers,  first  responders  and  military  personnel  against  smallpox 
in  preparation  for  a  possible  bioterrorist  attack.3  From  December  2002-January  2005, 
nearly  40,000  health  care  workers  and  first  responders  were  inoculated  against  smallpox 
as  well  as  more  than  730,000  US  military  personnel.5 

When  possible,  establishing  a  measure  of  reliability  in  self-reported  data  will  yield 
insight  into  potential  strengths  and  limitations  of  questionnaire  data.  Most  adult  vaccine 
studies  have  indicated  that  self-report  is  a  moderately  reliable  method  to  determine 
an  individual’s  vaccination  status,6'13  but  no  previous  study  to  date  has  investigated 
the  accuracy  of  self-reported  smallpox  vaccination.  Previous  studies  have  found  that 
self-report  of  adult  influenza  and  pneumococcal  vaccination  is  highly  sensitive  (70-100%) 
but  less  specific  (22-89%). 6,7,10,13  Using  limited  anthrax  vaccine  data,  the  sensitivity 
of  self-reported  anthrax  vaccination  among  1991  Gulf  War  veterans  was  73. 9%. 8  Among 
Millennium  Cohort  Study  members,  greater  than  substantial  agreement  (k  =  0.80) 
was  found  between  self-report  and  electronic  anthrax  vaccination  status.11  Two  previous 
studies  found  smallpox  vaccinees  were  accurately  able  to  self-report  their  reaction  to  the 
smallpox  vaccination.14,15  Using  data  from  the  Millennium  Cohort  Study,  the  purpose 
of  this  study  was  to  investigate  agreement  between  self-reported  smallpox  vaccination 
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and  electronic  vaccination  records  in  a  large,  population-based 
military  cohort.  Further,  this  study  examines  the  variation  in 
vaccination  data  concordance  as  it  pertains  to  subjective  and 
objective  measures  of  physical  and  mental  health. 

METHODS 

Study  population  and  data  sources.  The  Millennium  Cohort 
Study  was  launched  in  2001  to  gather  and  evaluate  population-based 
data  on  behavioral  and  occupational  risk  factors  related  to  military 
service  that  may  be  associated  with  adverse  health  outcomes.16,17 
A  randomly  selected  group  from  all  US  military  personnel  serving 
in  2000  was  invited  to  participate  in  the  first  panel.  Those  who 
had  been  deployed  to  Southwest  Asia,  Bosnia  or  Kosovo 
(January  1,  1998-September  1,  2000),  members  of  the  Reserve  or 
National  Guard  and  women  were  oversampled  to  ensure  sufficient 
power  to  detect  differences  in  these  smaller  population  subgroups. 
Of  the  77,047  enrolled  in  this  first  panel,  55,021  (71.4%) 
completed  the  follow-up  questionnaire  between  June  2004  and 
February  2006.  The  population  for  the  current  study  included 
Millennium  Cohort  Study  participants  with  complete  demo¬ 
graphic  and  military-specific  data  who  consented  at  baseline, 
participated  in  the  follow-up,  and  answered  the  smallpox  vaccine 
question  on  the  follow-up  questionnaire. 

The  Defense  Manpower  Data  Center  (DMDC)  provided  demo¬ 
graphic  and  military-specific  data  from  electronic  personnel  files, 
including  gender,  birth  date,  highest  education  level,  marital  status, 
race/ethnicity,  deployment  experience  (no  deployment  experience; 
Gulf  War,  Bosnia,  Kosovo  or  Southwest  Asia  deployment  experi¬ 
ence  before  the  Global  War  on  Terrorism  (GWOT);  or  deployment 
experience  in  support  of  GWOT  between  2001  and  2006), 
pay  grade,  service  component  (active  duty  or  Reserve/National 
Guard),  service  branch  (Army,  Air  Force,  Navy,  Coast  Guard 
or  Marine  Corps)  and  primary  military  occupations. 

Smallpox  vaccination  data.  The  following  question  was  included 
on  the  2004-2006  questionnaire  to  measure  smallpox  vaccination, 
“In  the  past  three  years  have  you  received  the  smallpox  vaccine?” 
The  questionnaire  did  not  include  information  about  the 
vaccine’s  administration  or  skin  reaction  at  the  injection  site. 
Electronic  smallpox  vaccination  data  were  obtained  from  the  Defense 
Enrollment  and  Eligibility  Reporting  System  at  DMDC  which 
maintains  an  electronic  database  of  all  vaccinations  given  to  military 
service  members.  A  participant  was  classified  as  vaccinated  in  the 
electronic  records  if  a  smallpox  vaccination  was  documented  in  the 
three  years  prior  to  completing  the  follow-up  questionnaire. 

Health  metrics.  The  baseline  and  follow-up  questionnaires 
of  the  Millennium  Cohort  Study  include  questions  to  assess  physical 
health,  mental  health,  and  other  health  outcomes.  Differences 
between  vaccination  concordance  groups  were  examined  as  they 
pertained  to  prior  hospitalization  and  subjective  health  metrics. 
Inpatient  data  were  used  to  investigate  differences  in  objective 
measures  of  health  using  hospitalization  as  the  outcome  of  interest 
among  active-duty  service  members.  Participants  were  classified 
as  hospitalized  if,  in  the  12  months  prior  to  completing  the  survey, 
they  were  hospitalized  for  any  cause,  excluding  the  following 
International  Classification  of  Diseases,  Ninth  Revision,  Clinical 
Modification  diagnosis  codes:  complications  of  pregnancy,  child¬ 
birth,  and  the  puerperium  (630-77),  congenital  anomalies 
(740-59),  and  certain  conditions  originating  in  the  perinatal 
period  (760-79).  Hospitalization  data  were  obtained  from  the 


computerized  databases  of  standardized  discharge  diagnoses  for 
hospitalizations  within  the  Military  Health  System  and  for 
hospitalizations  billed  to  the  Department  of  Defense  by  nonmili¬ 
tary  facilities.  These  databases  contain  hospitalization  summaries 
including  dates  of  admission  and  discharge,  and  up  to  eight 
individual  discharge  diagnoses  for  each  encounter,  which  are 
uniformly  coded  across  US  military  services.  The  Medical 
Outcomes  Study  Short  Form  36-Item  Health  Survey  for  Veterans 
(SF-36V),  a  modified  version  of  the  MOS  Short  Form  36  (SF-36) 
that  is  contained  within  the  Millennium  Cohort  questionnaire, 
was  used  to  compare  differences  in  physical  and  mental 
health.18'22  Like  the  SF-36,  the  SF-36V  uses  standardized  scoring 
algorithms  to  assess  eight  health  scales:  physical  functioning, 
role  limitations  caused  by  physical  problems,  bodily  pain,  general 
health,  vitality,  social  functioning,  role  limitations  caused  by 
emotional  problems  and  mental  health.  Higher  scores  are  associated 
with  better  health  status.  Unique  identifiers  were  used  to  link 
the  immunization,  demographic  and  hospitalization  data  to  the 
Millennium  Cohort  participants. 

Statistical  analysis.  A  descriptive  investigation  was  completed  to 
compare  self-reported  vaccination  status  and  electronic  vaccination 
records.  The  percentage  of  disagreement  between  self-reported  vacci¬ 
nation  and  electronic  documentation  of  smallpox  vaccination  was 
reported.  To  measure  the  degree  of  nonrandom  agreement  between 
self-reported  smallpox  vaccination  and  electronic  smallpox  vaccine 
records  the  kappa  statistic  was  used.23  A  K  between  0. 8-1.0  was 
defined  as  “greater  than  substantial  agreement,”  between  0.6-0. 8 
as  “substantial  agreement,”  between  0.4-0. 6  as  “moderate  agree¬ 
ment,”  between  0.2-0. 4  as  “fair  agreement,”  and  between  0.0-0. 2 
as  “slight  or  poor  agreement.”24 

To  assess  associations  between  prior  hospitalization  and 
vaccination  concordance  status  multivariable  logistic  regression  was 
used,  adjusting  for  all  demographic  and  military-related  variables. 
Analysis  of  variance  was  performed  to  investigate  the  association 
between  self-reported  health  status  (SF-36V)  and  vaccination 
concordance,  adjusting  for  all  demographic  and  military- related 
variables.  Vaccination  concordance  was  divided  into  four  categories: 
(1)  participants  who  reported  not  receiving  a  smallpox  vaccination 
with  concurrence  from  the  electronic  records,  (2)  participants  who 
reported  receiving  a  smallpox  vaccination  but  electronic  records  did 
not  reflect  vaccination,  (3)  participants  who  reported  not  receiving 
a  smallpox  vaccination  but  electronic  records  reflected  vaccination 
and  (4)  participants  who  reported  receiving  a  smallpox  vaccination 
with  concurrence  from  the  electronic  records. 

Regression  diagnostics,  including  examining  covariates  for  multi- 
collinearity  and  goodness  of  fit  test,  were  performed.  All  data  analyses 
were  completed  using  SAS  Version  9.1.3  (SAS  Institute,  Inc.,  Cary, 
North  Carolina). 

RESULTS 

This  study  included  54,066  Millennium  Cohort  participants 
who  completed  the  follow-up  questionnaire  between  June  2004- 
February  2006,  answered  the  smallpox  vaccination  question, 
and  had  complete  demographic  and  military-specific  data. 
The  majority  of  participants  self-reported  they  had  not  received 
a  smallpox  vaccination  in  the  three  years  prior  to  completing  the 
questionnaire  (n  =  34,859,  64.5%).  The  electronic  vaccination 
records  confirmed  33,614  (96.4%)  of  these  participants  had  not 
received  a  smallpox  vaccination.  A  smaller  percentage  of  participants 
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Table  1  Characteristics  of  Millennium  Cohort  Study  participants  by  smallpox  vaccination  status 


Smallpox  vaccination  status 


Study 

sample 

N  =  54,066 

Concordant^ 

unvaccinated 

N  =  33,614 

Discordant! 

self-report 

vaccinated 

n  =  7,537 

Discordant! 

electronic 

vaccinated 

N  =  1,245 

Concordant! 

vaccinated 

N  =  11,670 

p-value! 

Characteristic 

%* 

n  (%) 

n  (%) 

n  (%) 

n  (%) 

Gender 

Male 

73.4 

23,477  (69.8) 

5,935  (78.7) 

922  (74.1) 

9,331  (80.0) 

<.0001 

Female 

26.6 

10,137  (30.2) 

1,602  (21.3) 

323  (25.9) 

2,339  (20.0) 

Birth  Year 

Before  1 960 

24.5 

9,072  (27.0) 

2,123  (28.2) 

204  (16.4) 

1,849  (15.8) 

<.0001 

1960-1969 

40.6 

13,497  (40.2) 

3,144  (41.7) 

559  (44.9) 

4,723  (40.5) 

1970-1979 

30.8 

9,896  (29.4) 

1,923  (25.5) 

429  (34.5) 

4,408  (37.8) 

1  980  or  later 

4.1 

1,149  (3.4) 

347  (4.6) 

53  (4.3) 

690  (5.9) 

Education 

High  school  or  less 

45.6 

13,569  (40.4) 

3,780  (50.2) 

675  (54.2) 

6,621  (56.7) 

<.0001 

Some  college 

17.8 

6,942  (20.7) 

1,104  (14.7) 

207  (16.6) 

1,367  (1 1.7) 

Bachelor's  degree 

22.1 

7,603  (22.6) 

1,678  (22.3) 

227  (18.2) 

2,428  (20.8) 

Advanced  degree 

14.6 

5,500  (16.4) 

975  (12.9) 

136  (10.9) 

1,254  (10.8) 

Marital  status 

Not  married 

26.7 

8,871  (26.4) 

2,038  (27.0) 

301  (24.2) 

3,228  (27.7) 

0.009 

Married 

73.3 

24,743  (73.6) 

5,499  (73.0) 

944  (75.8) 

8,442  (72.3) 

Race/ethnicity 

White  non-Hispanic 

71.0 

24,209  (72.0) 

5,178  (68.7) 

829  (66.6) 

8,185  (70.1) 

<.0001 

Black  non-Hispanic 

12.1 

4,031  (12.0) 

1,032  (13.7) 

212  (17.0) 

1,279  (1 1.0) 

Other 

16.9 

5,374  (16.0) 

1,327  (17.6) 

204  (16.4) 

2,206  (18.9) 

Deployment  experience^ 

No  deployment 

46.3 

19,762  (58.8) 

3,635  (48.2) 

257  (20.6) 

1,398  (12.0) 

<.0001 

Pre-GWOT  deployment 

23.1 

9,875  (29.4) 

1,800  (23.9) 

142  (1 1.4) 

673  (5.8) 

2001-2006  GWOT  deployment 

15.3 

1,835  (5.5) 

1,135  (15.1) 

397  (31.9) 

4,923  (42.2) 

Both  pre-GWOT  and  GWOT  deployment  15.2 

2,142  (6.4) 

967  (12.8) 

449  (36.1) 

4,676  (40.1) 

Military  rank 

Enlisted 

70.8 

23,528  (70.0) 

5,434  (72.1) 

946  (76.0) 

8,374  (71.8) 

<.0001 

Officer 

29.2 

10,086  (30.0) 

2,103  (27.9) 

299  (24.0) 

3,296  (28.2) 

Service  component 

Reserve/Guard 

53.4 

18,560  (55.2) 

4,266  (56.6) 

476  (38.2) 

5,570  (47.7) 

<.0001 

Active  duty 

46.6 

15,054  (44.8) 

3,271  (43.4) 

769  (61.8) 

6,100  (52.3) 

Branch  of  service 

Army 

47.7 

15,035  (44.7) 

3,624  (48.1) 

554  (44.5) 

6,577  (56.4) 

<.0001 

Air  Force 

30.3 

10,540  (31.4) 

1,673  (22.2) 

51 1  (41.0) 

3,644  (31.2) 

Navy/Coast  Guard 

18.0 

6,752  (20.1) 

1,823  (24.2) 

151  (12.1) 

1,017  (8.7) 

Marine  Corps 

4.0 

1,287  (3.8) 

417  (5.5) 

29  (2.3) 

432  (3.7) 

Occupational  category 

Combat  specialists 

19.2 

5,977  (17.8) 

1,514  (20.1) 

235  (18.9) 

2,663  (22.8) 

<.0001 

Health  care  specialists 

1 1.4 

4,106  (12.2) 

526  (7.0) 

171  (13.7) 

1,352  (12.0) 

Functional  support 

22.9 

8,449  (25.1) 

1,820  (24.2) 

236  (19.0) 

1,879  (16.1) 

Others 

46.5 

15,082  (44.9) 

3,677  (48.8) 

603  (48.4) 

5,776  (49.5) 

^Percentages  rounded  and  may  not  sum  to  100.  ^Concordant  unvaccinated,  both  self-report  and  electronic  database  reflect  no  vaccination;  Discordant  self-report  vaccinated,  self-reported  vaccination  but  electronic 
database  reflects  no  vaccination;  Discordant  electronic  vaccinated,  self-reported  no  vaccination  but  electronic  database  reflects  vaccination,-  Concordant  vaccinated,  both  self-report  and  electronic  database  reflect 
vaccination.  ^Pearson  chi-square  p  value  examining  unadjusted  associations  between  smallpox  vaccination  status  and  individual  characteristics.  §Pre-GW0T  deployment,  deployed  to  the  1 991  Gulf  War  or  to  Bosnia,  Kosovo 
or  Southwest  Asia  between  January  1, 1998,  and  September  30, 2000;  2001-2006  GWOT  deployment,  deployed  in  support  of  the  Global  War  on  Terrorism  before  completion  of  follow-up  survey. 


reported  receiving  a  smallpox  vaccination  (n  =  19,207,  35.5%). 
Of  these  participants,  11,670  (60.8%)  had  an  electronic  record 
indicating  vaccination.  Among  all  participants,  8,782  (16.2%)  had 
discordant  results.  Most  participants  with  discordant  results  reported 
receiving  a  vaccination  even  though  electronic  records  indicated 


no  vaccination  (n  =  7,537;  13.9%).  There  were  1,245  (2.3%) 
with  electronic  documentation  of  smallpox  vaccination  who  reported 
not  receiving  the  vaccination. 

The  demographic  characteristics  of  these  four  vaccination 
groups  are  compared  in  Table  1.  A  greater  proportion  of  those 


www.landesbioscience.com 


Human  Vaccines 


247 


Validation  of  Smallpox  Vaccination  and  Measures  of  Health 


Table  2  Agreement*  between  self-reported  smallpox 
vaccination  and  electronic  records 

Electronic  record 

Vaccine 

No  vaccine 

Self-report 

Vaccine 

(N  =  12,915) 

(N  =  41,151) 

(N  =  19,207) 

No  vaccine 

1 1 ,670 

7,537 

(N  =  34,859) 

1,245 

33,614 

*Agreement  measures  for  self-report  are  as  follows:  k  =  0.62;  90.4%  of  participants  with  electronic 
documentation  of  smallpox  vaccination  self-reported  being  vaccinated;  81.7%  of  participants  with 
no  electronic  record  of  vaccination  self-reported  not  receiving  a  vaccination. 


who  had  an  electronically  confirmed  self-reported  vaccination 
(concordant  vaccinated)  were  male,  younger,  less  educated,  deployed 
in  support  of  GWOT,  active  duty,  Army  service  members  and 
combat  specialists  than  those  who  reported  not  receiving  a  vacci¬ 
nation,  which  was  confirmed  by  the  electronic  vaccination  record 
(concordant  unvaccinated).  A  larger  percentage  of  participants 
who  self-reported  vaccination  with  no  confirmation  by  electronic 
record  (discordant  self-reported  vaccinated)  were  older,  not  deployed 
in  support  of  GWOT,  Reserve/National  Guard,  in  the  Navy/Coast 
Guard  or  Marine  Corps  and  working  in  functional  support  roles  than 
were  concordant  vaccinated  participants. 

Of  all  participants  with  electronic  documentation  of  smallpox 
vaccination,  90%  self-reported  being  vaccinated;  and  of  all  partici¬ 
pants  with  no  electronic  record  of  vaccination,  82%  self-reported 
not  receiving  a  vaccination  (Table  2) .  The  overall  K  statistic  indicated 
substantial  agreement  (k  =  0.62).  While  most  of  the  K  statistics 
calculated  for  the  categories  of  demographic  characteristics  indi¬ 
cated  substantial  agreement  (k  =  0.6-0. 8),  participants  who  were 
born  before  I960  (k  =  0.51),  those  without  GWOT  deployment 
experience  (k  =  0.35),  Navy/Coast  Guard  service  members  (k  =  0.41) 
and  Marines  (k  =  0.53)  had  lower  agreement  scores. 


After  adjustment  for  gender,  age,  education,  marital  status, 
race/ethnicity,  deployment  experience,  pay  grade,  service  branch  and 
occupation,  hospitalization  rates  for  any  cause  in  the  year  prior 
to  survey  completion  were  not  significantly  different  between  the 
vaccination  groups  among  active-duty  service  members  (Table  3). 

The  SF-36V  mean  scores  for  the  eight  scales  were  adjusted 
for  gender,  age,  education,  marital  status,  race/ethnicity,  deploy¬ 
ment  experience,  pay  grade,  service  component,  service  branch 
and  occupation  (Table  3).  Overall,  the  SF-36V  adjusted  mean  scores 
were  relatively  high  for  each  vaccination  group,  indicating  good 
health.  However,  significantly  lower  adjusted  mean  scores  for  social 
functioning  and  role  limitations  caused  by  physical  and  emotional 
problems  were  found  among  participants  whose  electronic  records 
reflected  no  vaccination  but  who  self-reported  receiving  the  smallpox 
vaccination  when  compared  to  groups  with  concordant  status. 
The  adjusted  mean  score  for  vitality  was  slightly  but  significantly 
lower  among  participants  who  reported  they  did  not  receive 
a  vaccination  but  whose  electronic  records  reflected  a  vaccination. 
Additionally,  the  adjusted  mean  score  for  physical  functioning 
and  general  health  were  slightly  but  significantly  higher  among 
the  concordant  vaccinated  group  compared  to  the  concordant 
unvaccinated  group. 

DISCUSSION 

This  study  demonstrated  that  self-reported  smallpox  vaccination 
records  are  consistent  with  electronic  military  vaccination  records, 
with  84%  of  participants  having  identical  responses  from  the  two 
sources.  Agreement  levels,24  as  demostrated  by  the  kappa  statistic, 
suggest  substantial  agreement  between  electronic  and  self-reported 
smallpox  vaccination  (k  =  0.62).  While,  to  our  knowedge, 
no  previous  studies  have  investigated  the  validity  of  self- 
reported  smallpox  vaccination,  previous  studies  have  examined 
other  vaccinations  given  to  adults.  A  slightly  higher  agreement 
(k  =  0.69)  was  found  between  self  report  and  medical  records  for 
pneumococcal  vaccination  among  Australian  adults.6  Two  other 


Table  3  Adjusted  odds  of  prior  hospitalization  and  adjusted  means  of  SF-36V  health  scores 
among  Millennium  Cohort  participants*  by  smallpox  vaccination  status 


Smallpox  vaccination  status 


Concordantf 

Discordantf 

Discordantf 

Concordantf 

unvaccinated 

self-report 

electronic 

vaccinated 

Health  outcome 

OR  (95%  Cl)f 

vaccinated 

OR  (95%  Cl) 

vaccinated 

OR  (95%  Cl) 

OR  (95%  Cl) 

Any-cause  hospitalization§ 

1.00 

1.19  (0.97-1.47) 

0.88  (0.57-1.35) 

0.88  (0.71-1.10) 

SF-36VH 

Mean 

Mean 

Mean 

Mean 

Physical  functioning 

92.71 

92.4' 

92.0' 

93. 52 

Role  physical 

93.61 

92. 82 

93. 71'2 

94. 21 

Bodily  pain 

76. 31-2 

75. 62 

74. 91'2 

76. 61 

General  health 

76.91 

76.6' 

75.4' 

78. 22 

Social  functioning 

88. O’ 

86. 52 

87. 51'2 

88. 31 

Role  emotional 

94. O’ 

93. 22 

93. 91'2 

94. 21 

Mental  health 

78.61 

77. 72 

77. 71'2 

78.41'2 

Vitality 

58. 81-2 

58. 42 

56. 63 

59.41 

^Participants  whose  SF-36V  component  could  not  be  scored  due  to  insufficient  questionnaire  responses  were  removed  from  analyses  (n  varies  by  component;  maximum  removed:  446).  ^Concordant 
unvaccinated,  both  self-report  and  electronic  database  reflect  no  vaccination  (reference  group);  Discordant  self-report  vaccinated,  self-reported  vaccination  but  electronic  database  reflects  no  vaccina¬ 
tion;  Discordant  electronic  vaccinated,  self-reported  no  vaccination  but  electronic  database  reflects  vaccination;  Concordant  vaccinated,  both  self-report  and  electronic  database  reflect  vaccination.*  Odds 
ratio  (OR),  95%  confidence  interval  (Cl),  and  mean  are  adjusted  for  gender,  age,  education,  marital  status,  race/ethnicity,  deployment  experience,  pay  grade,  service  component  (SF-36V  only),  service 
branch,  and  occupation.^  Hospitalization  analysis  includes  active-duty  service  members  only  and  reflects  odds  of  any-cause  admission  in  the  year  prior  to  survey  response.11  SF-36V,  Medical  Outcomes 
Study  Short  Form  36-Item  Health  Survey  for  Veterans.  1-3  Different  numbers  indicate  vaccination  status  scores  that  are  significantly  different  from  each  other  (p  <  0.05)  using  Scheffe's  adjustment 
for  multiple  comparisons. 
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studies,  however,  conducted  among  elderly  Department  of  Veterans 
Affairs  patients  have  found  lower  agreement  (k  =  0.18  and  0.42) 
between  self-report  and  medical  record  documentation  for  pneu¬ 
mococcal  vaccination.7,10  Among  Millennium  Cohort  participants, 
agreement  between  self-report  and  electronic  records  for  anthrax 
vaccination  has  been  reported  higher  (k  =  0.80)  than  the  current 
study  found  for  smallpox  vaccination.11 

Consistent  with  previous  research,  a  higher  percentage  of  partici¬ 
pants  with  electronic  documentation  recalled  receiving  a  vaccination 
(90%)  than  participants  with  no  electronic  record  who  reported 
not  receiving  a  vaccination  (82%). 7,1 0,1 1,13  The  finding  of  a  lower 
agreement  compared  with  anthrax  vaccination  in  the  Millennium 
Cohort  is  interesting.  Self-reported  vaccinations  administered 
annually  or  in  succession  of  doses,  such  as  influenza  and  anthrax, 
tend  to  be  more  reliably  recalled  than  vaccinations  administered  in 
a  single  dose  or  infrequently,  like  pneumococcal  and  smallpox.6,7,10 
Primary  vaccination  against  anthrax  includes  a  six-dose  series  followed 
by  an  annual  booster,  whereas  smallpox  vaccination  consists  of  one 
dose  rarely  followed  by  a  booster.25,26  Even  though  it  is  adminis¬ 
tered  in  one  dose,  the  smallpox  vaccination  is  more  distinct  than 
other  vaccinations,  with  vaccinees  developing  a  skin  reaction  at  the 
injection  site  that  lasts  up  to  three  weeks  and  usually  leaves  a  perma¬ 
nent  scar.1  Therefore,  it  seems  likely  that  vaccinees  would  remember 
this  vaccination.  The  largest  source  of  disagreement  (86%),  however, 
is  from  those  who  stated  they  received  a  smallpox  vaccination  but 
did  not  have  electronic  documentation.  Furthermore,  the  smallpox 
question  on  the  survey  asks  if  the  vaccine  was  received  in  the 
past  three  years,  whereas  the  anthrax  question  asks  if  the  vaccine 
has  ever  been  received.  Since  the  smallpox  vaccine  was  routinely 
administered  to  children  in  the  United  States  until  1971  and  to 
military  recruits  until  1990,  it  is  possible  that  service  members  who 
received  a  smallpox  vaccine  during  this  time  may  have  incorrectly 
reported  receiving  this  vaccine  on  the  questionnaire.1  The  agreement 
for  those  who  were  born  before  I960  (k  =  0.51),  who  were  likely 
vaccinated  during  this  period,  is  low,  suggesting  they  may  have  only 
received  the  smallpox  vaccination  as  a  child  or  as  a  military  recruit. 
Furthermore,  anthrax  vaccination  may  be  more  easily  recalled  due 
to  controversy  surrounding  the  administration  of  the  multi-dose 
vaccine.27,28 

Individuals  who  self-reported  smallpox  vaccination  but  did  not 
have  electronic  confirmation  were  slightly  less  likely  to  have  deployed 
in  support  of  GWOT  and  more  likely  to  be  in  the  Navy/Coast 
Guard  and  work  in  functional  support  roles,  when  compared  with 
participants  whose  electronic  and  self-report  indicated  vaccination. 
Differences  among  the  service  branches  may  be  due  to  variations 
in  procedures  for  reporting  as  well  as  administering  vaccinations. 
For  example,  lower  agreement  (k  =  0.41)  among  Navy  personnel 
could  be  explained  by  delays  in  shipboard  vaccination  reporting 
to  the  electronic  system.  Those  with  no  deployment  experience 
in  support  of  GWOT  had  lower  agreement  (k  =  0.35)  between 
electronic  and  self-reported  vaccination  status  than  the  study 
population.  The  smallpox  vaccination  is  often  administered  in 
conjunction  with  deployment,  therefore  those  with  recent  deploy¬ 
ment  experience  may  recall  their  vaccination  more  easily  because 
they  were  able  to  associate  it  with  their  deployment.  The  lower 
agreement  among  those  not  deployed  in  support  of  GWOT 
may  also  be  associated  with  age,  since  those  deploying  tend  to 
be  younger.  Moreover,  the  older  group  of  nondeployers  may 
be  recalling  childhood  vaccinations.  The  lower  kappa  statistic 
among  those  not  recently  deployed,  however,  may  also  be  attributed 


to  lower  prevalence  of  vaccination  among  this  group  since  the  kappa 
statistic  is  dependent  on  the  true  prevalence  of  the  examined  variable 
and  tends  toward  zero  when  the  true  prevalence  is  very  high  or 
low.29 

While  there  were  some  significant  differences  in  SF-36V  scores 
between  the  concordance  groups,  there  was  no  apparent  trend 
of  reporting  lower  health  measures.  For  social  functioning  and  limi¬ 
tations  caused  by  physical  and  mental  problems,  participants  whose 
self-reported  vaccination  was  not  confirmed  by  electronic  record 
had  lower  adjusted  mean  scores,  indicating  slightly  worse  health 
than  those  with  concordant  electronic  and  self-reported  vaccination 
status.  Those  whose  vaccination  status  agreed  had  significantly 
higher  physical  functioning  and  general  health  adjusted  mean  scores 
compared  with  the  other  concordance  groups;  those  who  reported 
no  vaccination  but  had  a  vaccination  recorded  in  the  electronic 
records  had  a  lower  adjusted  mean  score  for  vitality  compared  with 
all  other  groups.  In  a  previous  study  among  Millennium  Cohort 
participants,  lower  adjusted  mean  scores  were  found  for  all  eight 
SF-36V  scales  among  participants  whose  self-reported  anthrax 
vaccination  was  not  confirmed  in  the  electronic  records.11  This  indi¬ 
cation  that  some  Millennium  Cohort  participants  may  report  more 
exposures  as  well  as  more  health  challenges  was  not  as  strongly  supported 
among  discordance  of  reporting  in  the  current  study.  However, 
consistent  with  the  previous  vaccination  study  using  Millennium 
Cohort  data,  there  were  no  significant  differences  in  the 
hospitalization  experience  between  the  smallpox  vaccination  groups 
for  the  one  year  prior  to  survey  completion.11 

In  the  unfortunate  event  of  a  smallpox  bioterrorist  attack, 
vaccine  doses  will  be  limited  in  supply,  and  time  will  be  of  critical 
importance.  Results  of  the  current  study  suggest  that  96%  of  those 
who  report  not  being  vaccinated  are  confirmed  unvaccinated  in  the 
electronic  database  whereas  6 1  %  of  those  who  self-report  vaccination 
have  electronic  documentation  of  vaccination.  While  the  smallpox 
vaccination  scar  may  be  used  to  identify  those  who  are  vaccinated, 
self-reported  vaccination  status  could  also  be  used  as  a  screening 
tool  to  correctly  identify  those  who  have  not  been  vaccinated. 
Though  some  individuals  may  be  revaccinated,  the  biggest  risk  of 
misclassification  and  possible  smallpox  infection  and  transmission, 
would  be  among  individuals  who  self-report  being  vaccinated 
who  have  not  truly  received  the  vaccination.  Resources  could  then 
be  better  used  to  verify  vaccination  within  medical  records 
or  immunization  data  for  this  smaller  group  who  self-report  receiving 
the  smallpox  vaccination. 

This  study  has  several  limitations.  The  study  population  consists 
of  a  sample  of  Millennium  Cohort  participants  and  may  not  be 
representative  of  the  military  population  in  general.  However, 
investigation  of  potential  biases  in  the  Millennium  Cohort  have 
found  a  well-representative  military  cohort  who  report  reliable  data 
and  who  are  not  influenced  to  participate  by  poor  health  prior 
to  enrollment  (Wells  T,  Naval  Health  Research  Center,  unpublished 
manuscript).11,17,30'35  Since  a  considerable  percentage  of  military 
service  members  received  a  smallpox  vaccine  starting  in  early  2003, 
the  dependence  of  the  kappa  statistic  on  the  true  prevalence  of  the 
examined  variable  should  not  substantially  affect  the  findings  of  this 
study.29  The  Millennium  Cohort  questionnaire  does  not  explain 
the  administration  process  or  typical  skin  reaction  of  the  smallpox 
vaccination.  Without  this  information  participants  may  find  it 
difficult  to  identify  and  report  which  vaccinations  they  have  received. 
Finally,  the  amount  of  incomplete  documentation  for  smallpox 
vaccination  in  the  database  remains  unknown.  While  medical 
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records  may  have  provided  additional  vaccination  history,  recent 
research  indicates  a  strong  level  of  agreement  (sensitivity  =  93.8%, 
specificity  =  87.0%)  between  electronically  maintained  anthrax 
vaccination  records  and  anthrax  vaccination  data  abstracted  from 
medical  charts.36 

Despite  these  limitations,  there  are  many  strengths  to  these 
analyses.  This  study  is  the  first  large,  population-based  study 
of  concordance  between  self-report  and  objective  smallpox  vaccination 
data.  The  unique  opportunity  to  compare  these  data  sources  allows 
insight  into  the  reliability  of  other  self-reported  measures  on  the 
questionnaire  and  is  important  for  future  longitudinal  assessment 
when  only  self-reported  data  are  available.  The  large  sample  size 
of  this  study  allows  for  robust  comparison  of  self-report  and  objective 
measures  of  smallpox  vaccination.  Further,  this  study  was  able  to  link 
both  objective  and  subjective  measures  of  morbidity  to  self-reported 
and  electronic  vaccination  status. 

Vaccination  against  biological  agents  is  an  important  component 
of  protection  against  bio  terrorist  attacks.  During  a  time  of  crisis 
or  bioterrorist  attack,  knowing  the  limitations  and  strengths  of 
self-reported  vaccination  data  is  vital  to  the  success  and  containment 
of  emergency  response  efforts.  Further,  understanding  the  limitations 
and  strengths  of  self-reported  data  in  research  allows  for  appropriate 
interpretation  of  results.  Therefore,  it  is  important  to  establish 
the  reliability  of  self-reported  data,  such  as  smallpox  vaccination, 
and  to  investigate  differences  in  reporting  based  on  health  metrics. 
This  analysis  found  substantial  agreement  between  objectively 
maintained  records  and  self-reported  smallpox  vaccination. 
Further,  no  substantial  differences  in  health  metrics  were  found 
among  the  small  number  of  those  whose  recall  of  vaccination  did 
not  concur  with  their  objective  records.  Overall  results  support 
that  self-report  of  smallpox  vaccination  is  a  reliable  measure  of  this 
important  exposure. 
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